Softmax Modeling of Piecewise Semantics in Arbitrary State Spaces for ‘Plug and Play’ Human-Robot Sensor Fusion

We describe ongoing work toward using human collaborators as providers of ‘soft sensor’ data, which can be formally combined with conventional ‘hard sensor’ data to augment robotic state estimation and model learning. Formal integration of robotic and human sensing can greatly improve the robustness of autonomous perception and decision making, especially in unstructured environments where uncertainties cannot be well-characterized in advance and must be modeled/adapted on the fly. This is particularly important for unmanned aerial system (UAS) applications such as large-scale surveillance [1], [2] and wilderness search and rescue [3], in which autonomous vehicles are subject to size, weight and power constraints that restrict onboard sensing, processing and communication abilities. Soft data integration in multi-robot teams can also allow human teammates to stay ‘in the loop’ without cognitively overloading them or undermining robotic autonomy: rather than exert high mental effort to provide direct commands, a human sensor can perform the cognitively easier task of providing helpful observations, which can be consumed at each robot’s discretion [4]. Soft data can be broadly related to either ‘abstract’ phenomena that cannot be measured by robotic sensors (e.g. labels for object categories and behaviors) or measurable dynamical physical states that must be monitored constantly (object position, velocity, attitude, temperature, size, mass, etc.) [5]. This work focuses on the latter, under the key assumption that humans are not oracles: as with any other sensor data, human observations are subject to errors, limitations and ambiguities that must be handled properly. As such, we aim to adapt widely used statistical sensor fusion and robotic state estimation algorithms, e.g. Kalman filters and the like, so that soft data can be exploited with minimal effort on the part of the robot or the human sensor. Refs. [6], [7], [8] were among the first to develop Bayesian fusion techniques allowing human sensors to directly ‘plug into’ robotic state estimation and perception algorithms. However, these works assume that humans report data the same way robots do, and thus greatly limit the flexibility of humanrobot communication. In the context of target tracking with extended Kalman filters, for instance, [6] assumes that humans provide numerical range and bearing measurement reports (‘The target is at range 10 m, bearing 45 degrees’). Ref. [9] showed how to model and fuse flexible semantic